Latency solutions

ABSTRACT

In processing a game scene for display, in one embodiment input controller position information from a host memory is provided directly to a graphics processor rather than first being processed by a 3D application in a host processor. This results in more direct and timely processing of position information and reduces the number of 3D processing pipeline steps the controller position information must pass through thus reducing the user&#39;s perceived latency between moving the input controller and seeing the displayed results. In another embodiment, the input controller position information is provided directly from an input controller to a graphics card or subsystem rather than first going through a host processor or memory. This results in even more direct and timely processing of position information by further reducing the number of 3D processing pipeline steps the controller position information must pass through thus further reducing the user&#39;s perceived latency.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 61/230,702 filed on Aug. 1, 2009 and entitled “LatencySolutions,” which is incorporated herein by reference in its entirety.

BACKGROUND

1. Field of the Invention

The field of this invention is in the area of computer input devices,including video game controllers, and more particularly toresponsiveness of systems receiving signals from such input devices.

2. Related Art

As the market for and variety of video games has increased, so has thedemand for video game controllers. Early controllers were typicallytrackballs in a fixed mount thus requiring the user or game player toremain fairly stationary while playing the video game. Seeking to givethe game player more freedom of movement as well as a greater range ofvideo game input control, still later controllers were handheld andlater, some incorporated accelerometers to determine controllermovement. Other, external detectors have also been used to detectcontroller movement. Such external detectors include ultrasound andoptical mechanisms that attempt to track movement of a controller in afield of view. A more advanced form of controller known in the artutilizes magnetic fields. Magnetic coils in these controllers detect thepresence of a magnetic field to determine position and orientation ofthe controller. Some of these controllers consist of two parts, acontroller part and a base unit part, which communicate with each otherwirelessly.

Regardless of the type of controller used, ideally there is minimaldelay between the user's movement and the resulting response on thedisplay screen. Any noticeable delay can interfere with their ability tointeract with the game. Avoiding such noticeable delays helps maintainthe user's illusion of being “in the game” thus ensuring a seamlessgaming experience

Modern'three-dimensional (3D) applications and systems, as used in manyvideo games, have a very deep pipeline of operations that result in thefinal image on the display screen. This pipeline allows the game systemto draw many frames in quick succession. However, the deeper thepipeline the longer it takes for a given frame to be created anddisplayed. This can interfere with the user's desired seamless gamingexperience.

What is needed, therefore, is an improved pipeline approach for 3Dapplications which reduces the time taken to process and display theresults of a user's movement of a game input controller.

SUMMARY

An improved system and method for generating a 3D scene is disclosedwhich takes input position information directly into a graphicsprocessor, either through a host memory or via a direct to 3D graphicscard or subsystem connector, for processing of the input positioninformation by the graphics processor in combination with scene elementsgenerated by a host processor thus eliminating a number of the 3Dpipeline processing steps for the position information thereby reducingthe latency between when a user moved the controller and sees thedisplayed result.

An exemplary method for generating a video game scene comprises:processing a video game application in a host processor to generate avideo game scene; receiving input controller position information into amemory of the host processor; retrieving, by a graphics processor, theinput controller position information from the memory of the hostprocessor; processing the generated game scene in the graphics processorusing the retrieved input controller position information to generate afinal video game scene; and outputting the final video game scene to adisplay device.

An exemplary system for generating a video game scene comprises: a hostprocessor configured to generate a video game scene; a host processormemory configured to store input controller position informationreceived from an input controller; and a graphics processor configuredto receive the generated video game scene, configured to retrieve thestored input controller position information, and configured to processthe generated video game scene using the retrieved input controllerposition information.

An exemplary method for generating a video game scene comprises:processing a video game application in a host processor to generate avideo game scene; receiving input controller position information froman input controller into a memory of a graphics processor without theinput controller position information first going through a memory ofthe host processor; processing the generated game scene in the graphicsprocessor using the input controller position information from thememory of the graphics processor to generate a final video game scene;and outputting the final video game scene to a display device.

An exemplary system for generating a video game scene comprises: a hostprocessor configured to generate a video game scene; a graphics cardconfigured to receive input controller position information from aninput controller; a graphics memory located on the graphics cardconfigured to store the received input controller position information;and a graphics processor located on the graphics card configured toreceive the generated video game scene, configured to retrieve thestored input controller position information from the graphics memory,and configured to process the generated video game scene using theretrieved input controller position information.

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 is a block diagram of a video game system as may be used invarious embodiments.

FIG. 2 is a block diagram of a 3D graphics pipeline showing a knownstandard latency approach, a software reduced latency solution and ahardware reduced latency solution according to various embodiments.

FIG. 3 is a block diagram of a video game system according to a hardwarereduced latency solution according to various embodiments.

FIG. 4 is a block diagram showing elements of a graphics card as may beimplemented in the video game system of FIG. 3.

DETAILED DESCRIPTION

Referring now to FIG. 1, a block diagram of a video game system as maybe used in embodiments of the present invention can be seen. Here, aGraphics Processing Unit (GPU) 101, a Central Processing Unit (CPU) 103,a Memory 107, and a Universal Serial Bus (USB) connector 109 are allcoupled via a Chipset 105. As is known in the art, a game applicationrunning on CPU 103 uses data and instructions received across Chipset105 from Memory 107 to generate graphics elements for the game beingplayed. These graphics elements are then sent across Chipset 105 to GPU101 for further processing so they can ultimately be shown on a displaydevice such as a television or computer screen (not shown).

Further, as is also known in the art, when a game user moves a gameinput controller (not shown) coupled to USB connector 109, controllerposition information is received by USB connector 109 from the inputcontroller and this controller position information passes throughChipset 105 and is placed in Memory 107. Later, when the gameapplication running on CPU 103 is ready to update the displayed scenefor the game, the controller position information is retrieved by CPU103 via Chipset 105 from Memory 107. The game application running on CPU103 then uses the controller position information to determine theposition or location of what is referred to herein as a key sceneelement (i.e., whatever display element is being directly controlled bythe motion of the controller, e.g., the game user's character and/orinstrument such as a bat, sword, gun, three dimensional (3D) cursor,etc.) in the game scene being generated. CPU 103 outputs the newlygenerated game scene primitives, including the user's character and/orinstrument, to GPU 101 via Chipset 105. GPU 101 then performs furthergraphics processing on the game scene primitives and, ultimately,outputs a final game scene for display on a display device (not shown).

This time period between when a user moves a game input controller andwhen the resulting change in displayed character and/or instrument isdisplayed is referred to herein as “latency.” Unfortunately, in theprior art, this latency can be long enough to cause a noticeable delaywhich can interfere with the user's ability to interact with the game.

In one embodiment of the present invention, known herein as a softwarereduced latency solution, the controller position information fromMemory 107 is provided directly to GPU 101, rather than first goingthrough CPU 103, which results in more direct and timely processing ofthis position information than occurs in the standard latency approachof the prior art. The number of pipeline steps the controller positioninformation must pass through is thereby reduced and since thecontroller position information does not have to first go through CPU103 to be included in the graphic scene generation by the gameapplication running on CPU 103, GPU 101 can receive newer or more recentcontroller position information when performing the further graphicsprocessing on the game scene that was generated by CPU 103. This resultsin a more direct and timely processing of the input controller positioninformation and reduces the user's perceived latency.

In another embodiment of the present invention, known herein as ahardware reduced latency solution, the controller position informationis provided directly from an input controller to a graphics card orsubsystem, including a graphics processor such as GPU 101 and a graphicsmemory (not shown), rather than first going through CPU 103 or Memory107. This results in an even more direct and timely processing of thisposition information than occurs in either the software reduced latencysolution or in the standard latency approach of the prior art. Thenumber of pipeline steps the controller position information must passthrough is thereby still further reduced and since the controllerposition information does not have to first go through CPU 103 or Memory107 to be included in the graphic scene processing running on GPU 101,GPU 101 can receive even newer or more recent controller positioninformation when performing the further graphics processing on the gamescene that was generated by CPU 103. This results in an even more directand timely processing of the input controller position information thusfurther reducing the user's perceived latency.

Referring now to FIG. 2, a block diagram can be seen of a 3D graphicspipeline showing the known standard latency approach, the softwarereduced latency solution and the hardware reduced latency solutionaccording to various embodiments.

A typical Personal Computer (PC)/game console/system (host) used forreal-time 3D applications generally performs the following sequence ofsteps to read user input and display its effect on a display screen.This sequence is known as a pipeline and it includes processing in theinput controller, the host Operating System (OS), the game applicationand the graphics subsystem as will be described.

In step 201, a game player user interacts with the game input controllerby, for example, swinging the controller like a baseball bat.

In step 202, the user's movement of the game input controller isrecognized by the internal electronics of the controller.

In step 203, the game input controller sends the new input controllerposition information wirelessly to a game input controller base unit orRadio Frequency (RF) Dongle.

In step 204, the game input controller base unit or RF Dongle receivesthe new input controller position information wirelessly from the gameinput controller.

In step 205, the game input controller base unit sends the new inputcontroller position information across a USB link to the host USBcontroller (e.g., USB 109 of FIG. 1).

In step 206, the host USB controller receives the new input controllerposition information and passes it to the host CPU (e.g., CPU 103 ofFIG. 1). The host OS, upon receiving the data and after some period oftime dependent upon the real-time nature of its processing, notifies aninput device driver of the newly arrived data.

In step 207, the input device driver copies the new input controllerposition information from a host CPU system buffer into its own memoryspace (e.g., in Memory 107). The new input controller positioninformation is now stored in memory ready to be queried by anapplication running on the host CPU.

Meanwhile, in step 209 the game application has been running on the hostCPU completing generating a current frame of graphics so it ignores thenew input controller position information for a while until it is readyto begin generating the next frame of graphics.

Once the current frame of graphics is complete, the game application'smain loop begins generating the next frame. Typically, it is at thebeginning of the loop that the game application queries the driver fornew input controller position information in step 210.

The game application then uses this new input controller positioninformation to compute the composition of the next game scene. Forexample, the position of the baseball bat for this next frame can now becomputed. This new input controller position information is used toposition the full 3D model of the baseball bat in the correct position.

Now that the baseball bat's position is known, in step 211 various otherphysical effects can be computed by the game application's physicsengine. For example, if the new baseball bat position caused a collisionwith a baseball, the baseball's trajectory can be computed. Computingthe new baseball trajectory can be quite sophisticated, taking intoaccount material properties of the baseball and baseball bat,atmospheric effects due to the air around the baseball, as well as thekinematic situation due to the baseball's spin and trajectory. Thespecific interaction to the motion is not important to note, but what issignificant is that this interaction is typically compute-intensive andis therefore done by the host's main processor, the CPU (e.g. CPU 103 ofFIG. 1).

Once the physics simulation is complete, the full scene composition iscomplete. At this point, every visual detail of the scene is set orfrozen and is exactly as it will be seen by the user when it is laterdisplayed on the graphics display. Thus, in step 212, the driver outputsthe scene information to memory (e.g., Memory 107) for use by thegraphics processor (e.g., GPU 101). Note, however, that the baseball batis in an already-old position queried from the device driver because theuser will likely have further moved the controller in the meantime. Assuch, it is not unlikely that still newer input controller positioninformation has arrived via USB and is available to the driver but sincethe game application only queries at the beginning of its main loop,that newer position information is not yet available to the gameapplication and will not (yet) be used. Furthermore, there remains someperiod of time left before this newly composed scene will be presentedto the user on the display, and several more recent instances of gamecontroller input position information will become available in themeantime, but they will all go unused in the new scene compositionsince, again, it is set or frozen at this point in time.

The game application now knows where all the scene components should beplaced and therefore completes the ‘application update’ phase of itsmain loop and begins the ‘graphics draw’ phase which consists ofcommunicating with the graphics card driver every detail of the scene.

In steps 212 through 213, what the graphics processor and associatedgraphics subsystem components do with the scene description from thispoint varies from host system to host system, but it can take some timefor the completed scene to complete being drawn. This time is typicallytaken by a graphics driver's optimization of the scene primitives intooptimal display lists or by pipelining in the graphics processor (e.g.,GPU 101 of FIG. 1).

In the past, graphics processors were only capable of drawing a sceneexactly as described by the game application. Every detail of objectplacement in the scene had to be computed by the host CPU (e.g., CPU 103of FIG. 1). Today, due to the complexities of computing dynamic geometrysuch as smoke, water or crashed car fenders, modern graphics processors(e.g., GPU 101 of FIG. 1) apply their highly-parallel computationabilities directly to geometric primitives. This means that the host CPU(e.g., CPU 103 of FIG. 1) can simply indicate that water is to be placedin some area of the scene and leave it up to the graphics processor tocompute every wave and splash. This computation is done via a speciallanguage that runs on the graphics processor. This is known asprogramming with shaders and examples of such languages include OpenGLShading Language (GLSL) and High Level Shader/Shading Language (HLSL).

In step 214, once the graphics processor is done drawing the scene, thatis, after completion of the ‘graphics draw phase,’ the scene is ready tobe displayed on the display screen. This can only happen when thegraphics display screen is ready to draw a next frame, so the screenimage may still wait a bit until the graphics display is ready for it.Even after being handed over to the graphics display, modern graphicsdisplays (e.g., televisions), may wait even longer before they actuallyshow the image.

This standard latency pipeline approach sequence commonly results inlatency for simple controller button presses of upwards of 200milliseconds. However, using the software reduced latency solution, thegame input controller position information bypasses the host CPU (e.g.,CPU 103 of FIG. 1) and, instead, is received and processed directly bythe graphics processor (e.g., GPU 101 of FIG. 1) for latency-criticalobjects (i.e., key scene elements) such as the baseball bat. This meansthat such objects can be positioned using more recent game inputcontroller position information in Step 212 as described above, therebyremoving the operations of steps 206-211 from the total latency. Thus,the total latency of the software reduced latency solution reduces thetotal latency by the difference in time between time t_(SL) (timeStandard Latency) and time t_(SRL) (time Software Reduced Latency), asshown in FIG. 2.

Note further that, in one embodiment, having the graphics processor(e.g., GPU 101 of FIG. 1) retrieve the game input controller positioninformation directly from memory (e.g., Memory 107 of FIG. 1) can beaccomplished by programming the graphics card driver to retrieve thisinformation. Such programming can operate by inserting retrieved newer(more recent) controller position information into the graphics scenegeometry as specified by the shader language or replacing older (lessrecent) controller position information previously placed in the gamescene graphics geometry by the host CPU (e.g., CPU 102) according to astandard latency approach processing.

Referring now to FIG. 3, a block diagram of a video game systemaccording to the hardware reduced latency solution according to variousembodiments of the present invention can be seen. Here, the same GPU101, CPU 103, Chipset 105 and Memory 107 as described with reference toFIGS. 1 and 2 can be seen. Also shown is USB 303 which is a USB graphicsgame port for direct coupling of a game input controller (not shown) toGPU 101. In the hardware reduced latency solution, game input controllerposition information is thus fed directly to GPU 101 without first goingthrough Chipset 105, being stored in Memory 107 or necessarily beingprocessed by CPU 103. Such direct coupling of game input controllerposition information to a graphics card or subsystem facilitates furtherreductions in the overall latency as will be described.

Referring now to FIG. 4, a block diagram showing elements of a graphicscard 401 as may be implemented in the video game system of FIG. 3 can beseen. Graphics processors (e.g., GPU 101 of FIGS. 1 and 3) are commonlyplaced on a physically separate circuit board from a host processor(e.g., CPU 103 of FIGS. 1 and 3). This separate circuit board iscommonly referred to as a graphics card because its main function isgraphics processing. It is to be understood that GPU 101 of FIGS. 1 and3 could likewise be located on such a graphics card despite not beingexplicitly shown as such in those figures. It is also to be understoodthat the graphics card components and functionality described hereincould be a graphics subsystem co-located with a host processor (e.g.,CPU 103 of FIGS. 1 and 3) and host memory (e.g., Memory 107 of FIGS. 1and 3) without departing from the spirit and scope of the presentinvention.

In the embodiment shown, graphics card 401 contains GPU 101 coupled toUSB 303, Firmware 305 and Memory 307. Firmware 305 is the programmablecontrolling logic for GPU 101 and Memory 407 is the working memory forGPU 101, as is known in the art. USB 303 is essentially the samegraphics game port as described with reference to FIG. 3 although, ofcourse, other known input communication mechanisms and approaches couldlikewise be used. As also shown, GPU 101 is in communication with achipset (e.g., Chipset 105 of FIGS. 1 and 3) of a host processor (e.g.,CPU 103 of FIGS. 1 and 3) using any known Input/Output (I/O) Busarchitecture such as PCI (Peripheral Component Interconnect) Express.

In the hardware reduced latency solution, USB 303 receives gamecontroller position information from a game controller coupled thereto(not shown). This game controller position information is stored inMemory 407 as directed by Firmware 405. Then, when GPU 101 is drawingthe scene to be displayed (e.g., in Step 213 of FIG. 2), Firmware 405retrieves this stored position information and provides it to GPU 101.This can be implemented as vertex shader extensions in Firmware 405 toretrieve the stored position information in a form known in the art as atransformation matrix representing the controller position.

The hardware reduced latency solution has the benefit of providing GPU101 with game input controller position information at the last possiblemoment in the game scene drawing calculations and with the most recentlyavailable game input controller position information since it did nothave to go through multiple other components and processing before beingaccessed and used. Using the hardware reduced latency solution thusbypasses pipeline Steps 206 through 212 of FIG. 2 thereby reducing thetotal latency by the difference in time between time t_(SL) (timeStandard Latency) and time t_(HRL) (time Hardware Reduced Latency).

It is to be noted that the game controller input position informationcan also be passed to the host CPU (e.g., CPU 103 of FIGS. 1 and 3), viaChipset 105 either directly from USB 303 or from Memory 407, so that thegame application running on the host CPU can also incorporate thisposition information into its graphics scene processing as needed ordesired.

It is to be understood that various alternative embodiments andsequences of operation are to be included in the scope of the presentinvention. It is to be understood that the present approach can also beutilized for some process other than operation of a video game. In suchevent, the apparatus and operations described herein are equallyapplicable to player or user interaction with such other process inessentially the same manner as described herein. As such, the presentinvention can be utilized with any computing system requiring usercontroller input.

The embodiments discussed herein are illustrative of the presentinvention. As these embodiments of the present invention are describedwith reference to illustrations, various modifications or adaptations ofthe methods and or specific structures described may become apparent tothose skilled in the art. All such modifications, adaptations, orvariations that rely upon the teachings of the present invention, andthrough which these teachings have advanced the art, are considered tobe within the spirit and scope of the present invention. Hence, thedescription and the drawing should not be considered in a limitingsense, as it is understood that the present invention is in no waylimited to only the embodiments illustrated.

What is claimed is:
 1. A method for generating a video game scenecomprising: processing a video game application in a host processor togenerate a video game scene; receiving input controller positioninformation from an input controller into a memory of the hostprocessor; retrieving, by a graphics processor, the input controllerposition information from the memory of the host processor; processingthe generated game scene in the graphics processor by incorporating theretrieved input controller position information to generate a finalvideo game scene without processing of the input controller positioninformation by the video game application in the host processor; andoutputting the final video game scene to a display device.
 2. The methodfor generating a video game scene of claim 1 wherein the received inputcontroller position information is in the form of a transformationmatrix.
 3. A system for generating a video game scene comprising: a hostprocessor configured to generate a video game scene; a host processormemory configured to store input controller position informationreceived from an input controller; and a graphics processor configuredto receive the generated video game scene, configured to retrieve thestored input controller position information, and configured to processthe generated video game scene by incorporating the retrieved inputcontroller position information into a final video game scene withoutprocessing of the input controller position information by the hostprocessor to generate the video game scene.
 4. The system for generatinga video game scene of claim 3 wherein the received input controllerposition information is in the form of a transformation matrix.
 5. Amethod for generating a video game scene comprising: processing a videogame application in a host processor to generate a video game scene;receiving input controller position information from an input controllerinto a memory of a graphics processor without the input controllerposition information first going through a memory of the host processor;processing the generated video game scene in the graphics processor byincorporating the input controller position information from the memoryof the graphics processor to generate a final video game scene withoutprocessing of the input controller position information by the videogame application in the host processor; and outputting the final videogame scene to a display device.
 6. The method for generating a videogame scene of claim 5 wherein the received input controller positioninformation is in the form of a transformation matrix.
 7. A system forgenerating a video game scene comprising: a host processor configured togenerate a video game scene; a graphics card configured to receive inputcontroller position information from an input controller; a graphicsmemory located on the graphics card configured to store the receivedinput controller position information; and a graphics processor locatedon the graphics card configured to receive the generated video gamescene, configured to retrieve the stored input controller positioninformation from the graphics memory, and configured to process thegenerated video game scene by incorporating the retrieved inputcontroller position information into a final video game scene withoutprocessing of the input controller position information by the hostprocessor to generate the video game scene.
 8. The system for generatinga video game scene of claim 7 wherein the received input controllerposition information is in the form of a transformation matrix.